Distributed Stochastic Optimization of the Regularized Risk

Many machine learning algorithms minimize a regularized risk, and stochastic optimization is widely used for this task. When working with massive data, it is desirable to perform stochastic optimization in parallel. Unfortunately, many existing stochastic algorithms cannot be parallelized efficiently. In this paper we show that one can rewrite the regularized risk minimization problem as an equivalent saddle-point problem, which is amenable for distributed stochastic optimization (DSO). We prove rates of convergence of our algorithm. DSO outperforms state-of-the-art algorithms when used for training linear support vector machines (SVMs) and logistic regression on the publicly available datasets.